{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predicting sentiment on the IMDB dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fr 27. Apr 17:29:14 CEST 2018\r\n"
     ]
    }
   ],
   "source": [
    "!date"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we show you how to train an RNN to classify movie review sentences. We mostly start from scratch, so that you should be able to plug in your own dataset without too much hassle. Furthermore, we explain some best practices with skorch, and how to perform a randomized hyper-parameter search."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import tarfile"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dstoolbox.transformers import Padder2d\n",
    "from dstoolbox.transformers import TextFeaturizer\n",
    "import numpy as np\n",
    "from scipy import stats\n",
    "from sklearn.datasets import load_files\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "from skorch import NeuralNetClassifier\n",
    "import torch\n",
    "from torch import nn\n",
    "F = nn.functional"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Constants"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "VOCAB_SIZE = 1000  # This is on the low end\n",
    "MAX_LEN = 50  # Texts are pretty long on average, this is on the low end\n",
    "USE_CUDA = True  # Set this to False if you don't want to use CUDA\n",
    "NUM_CV_STEPS = 10  # Number of randomized search steps to perform"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we need to install an additional package, dstoolbox, for this example to run:\n",
    "\n",
    "    $ pip install dstoolbox\n",
    "\n",
    "Also, download the IMDB dataset\n",
    "\n",
    "    $ wget http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Untar and unzip data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "if not os.path.exists('aclImdb'):\n",
    "    # unzip data if it does not exist\n",
    "    with tarfile.open('aclImdb_v1.tar.gz', 'r:gz') as f:\n",
    "        f.extractall()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Description of the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Large Movie Review Dataset v1.0\r\n",
      "\r\n",
      "Overview\r\n",
      "\r\n",
      "This dataset contains movie reviews along with their associated binary\r\n",
      "sentiment polarity labels. It is intended to serve as a benchmark for\r\n",
      "sentiment classification. This document outlines how the dataset was\r\n",
      "gathered, and how to use the files provided. \r\n"
     ]
    }
   ],
   "source": [
    "!head -n 8 aclImdb/README"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "@InProceedings{maas-EtAl:2011:ACL-HLT2011,\r\n",
      "  author    = {Maas, Andrew L.  and  Daly, Raymond E.  and  Pham, Peter T.  and  Huang, Dan  and  Ng, Andrew Y.  and  Potts, Christopher},\r\n",
      "  title     = {Learning Word Vectors for Sentiment Analysis},\r\n",
      "  booktitle = {Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies},\r\n",
      "  month     = {June},\r\n",
      "  year      = {2011},\r\n",
      "  address   = {Portland, Oregon, USA},\r\n",
      "  publisher = {Association for Computational Linguistics},\r\n",
      "  pages     = {142--150},\r\n",
      "  url       = {http://www.aclweb.org/anthology/P11-1015}\r\n",
      "}\r\n",
      "\r\n",
      "References\r\n",
      "\r\n",
      "Potts, Christopher. 2011. On the negativity of negation. In Nan Li and\r\n",
      "David Lutz, eds., Proceedings of Semantics and Linguistic Theory 20,\r\n",
      "636-659.\r\n",
      "\r\n",
      "Contact\r\n",
      "\r\n",
      "For questions/comments/corrections please contact Andrew Maas\r\n",
      "amaas@cs.stanford.edu\r\n"
     ]
    }
   ],
   "source": [
    "!tail -n 22 aclImdb/README"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Read in data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = load_files('aclImdb/train/', categories=['pos', 'neg'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['data', 'filenames', 'target_names', 'target', 'DESCR'])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Only minimal data transformation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We mostly leave the data as is; for better reults, we should for instance remove markup, but we leave this out for brevity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "X, y = dataset['data'], dataset['target']\n",
    "X = np.asarray([x.decode() for x in X])  # decode from bytes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A peak at the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Target: pos\n",
      "Zero Day leads you to think, even re-think why two boys/young men would do what they did - commit mutual suicide via slaughtering their classmates. It captures what must be beyond a bizarre mode of being for two humans who have decided to withdraw from common civility in order to define their own/mutual world via coupled destruction.<br /><br />It is not a perfect movie but given what money/time the filmmaker and actors had - it is a remarkable product. In terms of explaining the motives and actions of the two young suicide/murderers it is better than 'Elephant' - in terms of being a film that gets under our 'rationalistic' skin it is a far, far better film than almost anything you are likely to see. <br /><br />Flawed but honest with a terrible honesty.\n",
      "\n",
      "Target: neg\n",
      "Words can't describe how bad this movie is. I can't explain it by writing only. You have too see it for yourself to get at grip of how horrible a movie really can be. Not that I recommend you to do that. There are so many clichés, mistakes (and all other negative things you can imagine) here that will just make you cry. To start with the technical first, there are a LOT of mistakes regarding the airplane. I won't list them here, but just mention the coloring of the plane. They didn't even manage to show an airliner in the colors of a fictional airline, but instead used a 747 painted in the original Boeing livery. Very bad. The plot is stupid and has been done many times before, only much, much better. There are so many ridiculous moments here that i lost count of it really early. Also, I was on the bad guys' side all the time in the movie, because the good guys were so stupid. \"Executive Decision\" should without a doubt be you're choice over this one, even the \"Turbulence\"-movies are better. In fact, every other movie in the world is better than this one.\n",
      "\n",
      "Target: pos\n",
      "Everyone plays their part pretty well in this \"little nice movie\". Belushi gets the chance to live part of his life differently, but ends up realizing that what he had was going to be just as good or maybe even better. The movie shows us that we ought to take advantage of the opportunities we have, not the ones we do not or cannot have. If U can get this movie on video for around $10, it´d be an investment!\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for text, target in zip(X[:3], y):\n",
    "    print(\"Target: {}\".format(dataset['target_names'][target]))\n",
    "    print(text)\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transformation steps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are many possible ways to transform our data so that we can pass it to our neural net. What we effectively need is to transform a list of strings to an array of indices, where each row corresponds to one sample, and in each row, each int represents a specific word.\n",
    "\n",
    "Below we show one way to achieve this using `TextFeaturizer` from dstoolbox (more information in this [notebook](https://nbviewer.jupyter.org/github/ottogroup/dstoolbox/blob/master/notebooks/Examples_transformers.ipynb#TextFeaturizer)). This transformer relies heavily on sklearn's `CountVectorizer`. By using sklearn under the hood instead of rolling our own transformation code, we gain the following benefits:\n",
    "\n",
    "* battle-tested, (mostly) bug free code\n",
    "* since it is an sklearn transformer, we can put it into a `Pipeline`\n",
    "* many parameters for us to test in a hyper-parameter search\n",
    "\n",
    "For more on the last point, see the section about randomized search."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we have to solve a small problem, namely that sentences have different number of words. This results in a heterogenous array but we need a homogeneous array. With the help of dstoolbox's `Padder2d`, we get this functionality in an sklearn transformer. (Note: We set `pad_value=VOCAB_SIZE` to give the padded value a unique index, since the other indices will range from 0 to VOCAB_SIZE-1)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Putting all of this together, these are the transformation steps in the pipeline before the actual neural net:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "steps = [\n",
    "    ('to_idx', TextFeaturizer(max_features=VOCAB_SIZE)),\n",
    "    ('pad', Padder2d(max_len=MAX_LEN, pad_value=VOCAB_SIZE, dtype=int)),\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is what the output looks like so far:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[220,  48, 104, 217, 190, 186,  63, 156, 186, 207, 193,  29, 218,\n",
       "        117, 215,  57, 205, 184,  54,  43, 129, 173, 199, 169, 181,  39,\n",
       "        102,  35, 205, 128,  19,  26,  27, 120, 133,  23,  76, 193,  95,\n",
       "        206,  87,  49, 190, 210,  77,  44,  38,  98, 140, 190],\n",
       "       [213,  33,  52,  94,  18, 187, 124, 101,  33,  67, 102,  32, 216,\n",
       "        137, 217,  87, 191, 163, 102,  76, 219, 190,  78,  17,  83, 133,\n",
       "         94,  93, 124, 158,  33,  19, 132, 179, 159, 217, 190,  57, 179,\n",
       "        183,  14, 170, 115,  40, 119,  12,   8, 142, 130, 185],\n",
       "       [ 65, 151, 181, 148, 153, 203,  98, 187, 108, 131, 124,  24,  79,\n",
       "        180,  36, 190, 109, 148, 133,  90, 105,  56,  31,  62, 195, 157,\n",
       "        179, 205,  88,  85, 201,  81, 190,  19, 103,  16,  82, 139, 116,\n",
       "         63,  25, 180, 124, 166, 196, 179, 202, 143, 190, 174]])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Pipeline(steps).fit_transform(X[:3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As desired, we have a homogeneous array of indices, exactly what we need."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The RNN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We define a rather simple RNN with just embeddings, a recurrent layer, and an output layer. To be later able to test all hyper-parameters, we make sure to pass them to the `__init__` of our pytorch module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "class RNNClassifier(nn.Module):\n",
    "    def __init__(\n",
    "            self,\n",
    "            embedding_dim=128,\n",
    "            rec_layer_type='lstm',\n",
    "            num_units=128,\n",
    "            num_layers=2,\n",
    "            dropout=0,\n",
    "    ):\n",
    "        super().__init__()\n",
    "        self.embedding_dim = embedding_dim\n",
    "        self.rec_layer_type = rec_layer_type.lower()\n",
    "        self.num_units = num_units\n",
    "        self.num_layers = num_layers\n",
    "        self.dropout = dropout\n",
    "\n",
    "        self.emb = nn.Embedding(VOCAB_SIZE + 1, embedding_dim=self.embedding_dim)\n",
    "        \n",
    "        rec_layer = {'lstm': nn.LSTM, 'gru': nn.GRU}[self.rec_layer_type]\n",
    "        # We have to make sure that the recurrent layer is batch_first,\n",
    "        # since sklearn assumes the batch dimension to be the first\n",
    "        self.rec = rec_layer(\n",
    "            self.embedding_dim, self.num_units, num_layers=num_layers, batch_first=True)\n",
    "\n",
    "        self.output = nn.Linear(self.num_units, 2)\n",
    "\n",
    "    def forward(self, X):\n",
    "        embeddings = self.emb(X)\n",
    "        # from the recurrent layer, only take the activities from the last sequence step\n",
    "        if self.rec_layer_type == 'gru':\n",
    "            _, rec_out = self.rec(embeddings)\n",
    "        else:\n",
    "            _, (rec_out, _) = self.rec(embeddings)\n",
    "        rec_out = rec_out[-1]  # take output of last RNN layer\n",
    "        drop = F.dropout(rec_out, p=self.dropout)\n",
    "        # Remember that the final non-linearity should be softmax, so that our predict_proba\n",
    "        # method outputs actual probabilities!\n",
    "        out = F.softmax(self.output(drop), dim=-1)\n",
    "        return out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We wrap the pytorch module into a skorch `NeuralNetClassifier`, since we are dealing with a binary classification task, and append the step to our transformation steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "steps.append(\n",
    "    ('net', NeuralNetClassifier(\n",
    "        RNNClassifier,\n",
    "        device=('cuda' if USE_CUDA else 'cpu'),\n",
    "        max_epochs=5,\n",
    "        lr=0.01,\n",
    "        optimizer=torch.optim.RMSprop,\n",
    "    ))\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are good to go:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe = Pipeline(steps)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 28.8 s, sys: 5.79 s, total: 34.6 s\n",
      "Wall time: 34.6 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "     steps=[('to_idx', TextFeaturizer(analyzer='word', binary=False, decode_error='strict',\n",
       "        dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',\n",
       "        lowercase=True, max_df=1.0, max_features=1000, min_df=1,\n",
       "        ngram_range=(1, 1), preprocessor=None, stop_words=None,\n",
       "        stri...ayers=2, batch_first=True)\n",
       "    (output): Linear(in_features=128, out_features=2, bias=True)\n",
       "  ),\n",
       "))])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%time pipe.fit(X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Randomized search"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The results above were already okay, but we have many hyper-parameters, so of course we would like to know which ones are best. Therefore, we perform a randomized search. For those not aware, a randomized search is like a grid search, but instead of testing the parameters systematically, they are drawn randomly from a distribution. In practice, compared to grid search, randomized search will find you better parameter values in a shorter amount of time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the randomized search, we turn off the net's verbosity to not clutter the notebook. Also, we set `train_split=None`. This is because we don't need an internal train/valid split, given that sklearn's `RandomizedSearchCV` already takes care of cross-validation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "     steps=[('to_idx', TextFeaturizer(analyzer='word', binary=False, decode_error='strict',\n",
       "        dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',\n",
       "        lowercase=True, max_df=1.0, max_features=1000, min_df=1,\n",
       "        ngram_range=(1, 1), preprocessor=None, stop_words=None,\n",
       "        stri...ayers=2, batch_first=True)\n",
       "    (output): Linear(in_features=128, out_features=2, bias=True)\n",
       "  ),\n",
       "))])"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe.set_params(net__verbose=0, net__train_split=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we would like to set the hyper-parameter range to test. With randomized search, we can either specify a list (mostly for discrete variables) or a `scipy.stats` distribution, from which sklearn will sample automatically.\n",
    "\n",
    "As we can see below, we can extend the randomized search to not only cover the parameters we defined for our RNN, but also to cover the way we construct our vocabulary using `TextFeaturizer`. This is why we said earlier that we should use it instead of implementing it ourselves. As shown below, we test:\n",
    "\n",
    "* stop_words: whether to remove english stop words or not\n",
    "* lowercase: whether to turn all words lower-cased or not\n",
    "* ngram_range: whether to use word uni-grams or bi-grams\n",
    "\n",
    "We could also easily switch from words to characters by setting `analyzer='char'`, but then we would probably need longer sequences.\n",
    "\n",
    "Additionally, we test some hyper-parameters on the RNN module itself (e.g. LSTM vs GRU) and on the skorch `NeuralNetClassifier` (e.g. `max_epochs`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "params = {\n",
    "    'to_idx__stop_words': ['english', None],\n",
    "    'to_idx__lowercase': [False, True],\n",
    "    'to_idx__ngram_range': [(1, 1), (2, 2)],\n",
    "    'net__module__embedding_dim': stats.randint(32, 256 + 1),\n",
    "    'net__module__rec_layer_type': ['gru', 'lstm'],\n",
    "    'net__module__num_units': stats.randint(32, 256 + 1),\n",
    "    'net__module__num_layers': [1, 2, 3],\n",
    "    'net__module__dropout': stats.uniform(0, 0.9),\n",
    "    'net__lr': [10**(-stats.uniform(1, 5).rvs()) for _ in range(NUM_CV_STEPS)],\n",
    "    'net__max_epochs': [5, 10],\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We define our randomized search and start fitting.\n",
    "\n",
    "For demonstration purposes, we perform only a low number of iterations (10) and only fit on the first 5000 samples. Of course, with more time, we should use more steps and include all samples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "search = RandomizedSearchCV(\n",
    "    pipe, params, n_iter=NUM_CV_STEPS, verbose=2, refit=False, scoring='accuracy', cv=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 10 candidates, totalling 30 fits\n",
      "[CV] net__lr=0.000394032773719, net__max_epochs=5, net__module__dropout=0.792428300327, net__module__embedding_dim=254, net__module__num_layers=3, net__module__num_units=206, net__module__rec_layer_type=lstm, to_idx__lowercase=False, to_idx__ngram_range=(2, 2), to_idx__stop_words=english \n",
      "[CV]  net__lr=0.000394032773719, net__max_epochs=5, net__module__dropout=0.792428300327, net__module__embedding_dim=254, net__module__num_layers=3, net__module__num_units=206, net__module__rec_layer_type=lstm, to_idx__lowercase=False, to_idx__ngram_range=(2, 2), to_idx__stop_words=english, total=  13.6s\n",
      "[CV] net__lr=0.000394032773719, net__max_epochs=5, net__module__dropout=0.792428300327, net__module__embedding_dim=254, net__module__num_layers=3, net__module__num_units=206, net__module__rec_layer_type=lstm, to_idx__lowercase=False, to_idx__ngram_range=(2, 2), to_idx__stop_words=english \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:   15.0s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  net__lr=0.000394032773719, net__max_epochs=5, net__module__dropout=0.792428300327, net__module__embedding_dim=254, net__module__num_layers=3, net__module__num_units=206, net__module__rec_layer_type=lstm, to_idx__lowercase=False, to_idx__ngram_range=(2, 2), to_idx__stop_words=english, total=  13.4s\n",
      "[CV] net__lr=0.000394032773719, net__max_epochs=5, net__module__dropout=0.792428300327, net__module__embedding_dim=254, net__module__num_layers=3, net__module__num_units=206, net__module__rec_layer_type=lstm, to_idx__lowercase=False, to_idx__ngram_range=(2, 2), to_idx__stop_words=english \n",
      "[CV]  net__lr=0.000394032773719, net__max_epochs=5, net__module__dropout=0.792428300327, net__module__embedding_dim=254, net__module__num_layers=3, net__module__num_units=206, net__module__rec_layer_type=lstm, to_idx__lowercase=False, to_idx__ngram_range=(2, 2), to_idx__stop_words=english, total=  13.3s\n",
      "[CV] net__lr=0.000394032773719, net__max_epochs=5, net__module__dropout=0.518351846001, net__module__embedding_dim=42, net__module__num_layers=3, net__module__num_units=75, net__module__rec_layer_type=gru, to_idx__lowercase=True, to_idx__ngram_range=(1, 1), to_idx__stop_words=None \n",
      "[CV]  net__lr=0.000394032773719, net__max_epochs=5, net__module__dropout=0.518351846001, net__module__embedding_dim=42, net__module__num_layers=3, net__module__num_units=75, net__module__rec_layer_type=gru, to_idx__lowercase=True, to_idx__ngram_range=(1, 1), to_idx__stop_words=None, total=   4.6s\n",
      "[CV] net__lr=0.000394032773719, net__max_epochs=5, net__module__dropout=0.518351846001, net__module__embedding_dim=42, net__module__num_layers=3, net__module__num_units=75, net__module__rec_layer_type=gru, to_idx__lowercase=True, to_idx__ngram_range=(1, 1), to_idx__stop_words=None \n",
      "[CV]  net__lr=0.000394032773719, net__max_epochs=5, net__module__dropout=0.518351846001, net__module__embedding_dim=42, net__module__num_layers=3, net__module__num_units=75, net__module__rec_layer_type=gru, to_idx__lowercase=True, to_idx__ngram_range=(1, 1), to_idx__stop_words=None, total=   4.6s\n",
      "[CV] net__lr=0.000394032773719, net__max_epochs=5, net__module__dropout=0.518351846001, net__module__embedding_dim=42, net__module__num_layers=3, net__module__num_units=75, net__module__rec_layer_type=gru, to_idx__lowercase=True, to_idx__ngram_range=(1, 1), to_idx__stop_words=None \n",
      "[CV]  net__lr=0.000394032773719, net__max_epochs=5, net__module__dropout=0.518351846001, net__module__embedding_dim=42, net__module__num_layers=3, net__module__num_units=75, net__module__rec_layer_type=gru, to_idx__lowercase=True, to_idx__ngram_range=(1, 1), to_idx__stop_words=None, total=   4.6s\n",
      "[CV] net__lr=1.38125857601e-06, net__max_epochs=5, net__module__dropout=0.757247507278, net__module__embedding_dim=70, net__module__num_layers=3, net__module__num_units=74, net__module__rec_layer_type=lstm, to_idx__lowercase=False, to_idx__ngram_range=(1, 1), to_idx__stop_words=english \n",
      "[CV]  net__lr=1.38125857601e-06, net__max_epochs=5, net__module__dropout=0.757247507278, net__module__embedding_dim=70, net__module__num_layers=3, net__module__num_units=74, net__module__rec_layer_type=lstm, to_idx__lowercase=False, to_idx__ngram_range=(1, 1), to_idx__stop_words=english, total=   4.6s\n",
      "[CV] net__lr=1.38125857601e-06, net__max_epochs=5, net__module__dropout=0.757247507278, net__module__embedding_dim=70, net__module__num_layers=3, net__module__num_units=74, net__module__rec_layer_type=lstm, to_idx__lowercase=False, to_idx__ngram_range=(1, 1), to_idx__stop_words=english \n",
      "[CV]  net__lr=1.38125857601e-06, net__max_epochs=5, net__module__dropout=0.757247507278, net__module__embedding_dim=70, net__module__num_layers=3, net__module__num_units=74, net__module__rec_layer_type=lstm, to_idx__lowercase=False, to_idx__ngram_range=(1, 1), to_idx__stop_words=english, total=   4.6s\n",
      "[CV] net__lr=1.38125857601e-06, net__max_epochs=5, net__module__dropout=0.757247507278, net__module__embedding_dim=70, net__module__num_layers=3, net__module__num_units=74, net__module__rec_layer_type=lstm, to_idx__lowercase=False, to_idx__ngram_range=(1, 1), to_idx__stop_words=english \n",
      "[CV]  net__lr=1.38125857601e-06, net__max_epochs=5, net__module__dropout=0.757247507278, net__module__embedding_dim=70, net__module__num_layers=3, net__module__num_units=74, net__module__rec_layer_type=lstm, to_idx__lowercase=False, to_idx__ngram_range=(1, 1), to_idx__stop_words=english, total=   4.6s\n",
      "[CV] net__lr=0.00159506240923, net__max_epochs=10, net__module__dropout=0.512656565331, net__module__embedding_dim=126, net__module__num_layers=3, net__module__num_units=139, net__module__rec_layer_type=lstm, to_idx__lowercase=False, to_idx__ngram_range=(1, 1), to_idx__stop_words=english \n",
      "[CV]  net__lr=0.00159506240923, net__max_epochs=10, net__module__dropout=0.512656565331, net__module__embedding_dim=126, net__module__num_layers=3, net__module__num_units=139, net__module__rec_layer_type=lstm, to_idx__lowercase=False, to_idx__ngram_range=(1, 1), to_idx__stop_words=english, total=  13.5s\n",
      "[CV] net__lr=0.00159506240923, net__max_epochs=10, net__module__dropout=0.512656565331, net__module__embedding_dim=126, net__module__num_layers=3, net__module__num_units=139, net__module__rec_layer_type=lstm, to_idx__lowercase=False, to_idx__ngram_range=(1, 1), to_idx__stop_words=english \n",
      "[CV]  net__lr=0.00159506240923, net__max_epochs=10, net__module__dropout=0.512656565331, net__module__embedding_dim=126, net__module__num_layers=3, net__module__num_units=139, net__module__rec_layer_type=lstm, to_idx__lowercase=False, to_idx__ngram_range=(1, 1), to_idx__stop_words=english, total=  13.5s\n",
      "[CV] net__lr=0.00159506240923, net__max_epochs=10, net__module__dropout=0.512656565331, net__module__embedding_dim=126, net__module__num_layers=3, net__module__num_units=139, net__module__rec_layer_type=lstm, to_idx__lowercase=False, to_idx__ngram_range=(1, 1), to_idx__stop_words=english \n",
      "[CV]  net__lr=0.00159506240923, net__max_epochs=10, net__module__dropout=0.512656565331, net__module__embedding_dim=126, net__module__num_layers=3, net__module__num_units=139, net__module__rec_layer_type=lstm, to_idx__lowercase=False, to_idx__ngram_range=(1, 1), to_idx__stop_words=english, total=  13.5s\n",
      "[CV] net__lr=0.0526534134901, net__max_epochs=10, net__module__dropout=0.532837738145, net__module__embedding_dim=226, net__module__num_layers=1, net__module__num_units=212, net__module__rec_layer_type=lstm, to_idx__lowercase=False, to_idx__ngram_range=(2, 2), to_idx__stop_words=english \n",
      "[CV]  net__lr=0.0526534134901, net__max_epochs=10, net__module__dropout=0.532837738145, net__module__embedding_dim=226, net__module__num_layers=1, net__module__num_units=212, net__module__rec_layer_type=lstm, to_idx__lowercase=False, to_idx__ngram_range=(2, 2), to_idx__stop_words=english, total=  12.8s\n",
      "[CV] net__lr=0.0526534134901, net__max_epochs=10, net__module__dropout=0.532837738145, net__module__embedding_dim=226, net__module__num_layers=1, net__module__num_units=212, net__module__rec_layer_type=lstm, to_idx__lowercase=False, to_idx__ngram_range=(2, 2), to_idx__stop_words=english \n",
      "[CV]  net__lr=0.0526534134901, net__max_epochs=10, net__module__dropout=0.532837738145, net__module__embedding_dim=226, net__module__num_layers=1, net__module__num_units=212, net__module__rec_layer_type=lstm, to_idx__lowercase=False, to_idx__ngram_range=(2, 2), to_idx__stop_words=english, total=  12.8s\n",
      "[CV] net__lr=0.0526534134901, net__max_epochs=10, net__module__dropout=0.532837738145, net__module__embedding_dim=226, net__module__num_layers=1, net__module__num_units=212, net__module__rec_layer_type=lstm, to_idx__lowercase=False, to_idx__ngram_range=(2, 2), to_idx__stop_words=english \n",
      "[CV]  net__lr=0.0526534134901, net__max_epochs=10, net__module__dropout=0.532837738145, net__module__embedding_dim=226, net__module__num_layers=1, net__module__num_units=212, net__module__rec_layer_type=lstm, to_idx__lowercase=False, to_idx__ngram_range=(2, 2), to_idx__stop_words=english, total=  12.8s\n",
      "[CV] net__lr=1.38125857601e-06, net__max_epochs=5, net__module__dropout=0.880713835579, net__module__embedding_dim=171, net__module__num_layers=1, net__module__num_units=118, net__module__rec_layer_type=lstm, to_idx__lowercase=True, to_idx__ngram_range=(2, 2), to_idx__stop_words=None \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  net__lr=1.38125857601e-06, net__max_epochs=5, net__module__dropout=0.880713835579, net__module__embedding_dim=171, net__module__num_layers=1, net__module__num_units=118, net__module__rec_layer_type=lstm, to_idx__lowercase=True, to_idx__ngram_range=(2, 2), to_idx__stop_words=None, total=   6.3s\n",
      "[CV] net__lr=1.38125857601e-06, net__max_epochs=5, net__module__dropout=0.880713835579, net__module__embedding_dim=171, net__module__num_layers=1, net__module__num_units=118, net__module__rec_layer_type=lstm, to_idx__lowercase=True, to_idx__ngram_range=(2, 2), to_idx__stop_words=None \n",
      "[CV]  net__lr=1.38125857601e-06, net__max_epochs=5, net__module__dropout=0.880713835579, net__module__embedding_dim=171, net__module__num_layers=1, net__module__num_units=118, net__module__rec_layer_type=lstm, to_idx__lowercase=True, to_idx__ngram_range=(2, 2), to_idx__stop_words=None, total=   6.5s\n",
      "[CV] net__lr=1.38125857601e-06, net__max_epochs=5, net__module__dropout=0.880713835579, net__module__embedding_dim=171, net__module__num_layers=1, net__module__num_units=118, net__module__rec_layer_type=lstm, to_idx__lowercase=True, to_idx__ngram_range=(2, 2), to_idx__stop_words=None \n",
      "[CV]  net__lr=1.38125857601e-06, net__max_epochs=5, net__module__dropout=0.880713835579, net__module__embedding_dim=171, net__module__num_layers=1, net__module__num_units=118, net__module__rec_layer_type=lstm, to_idx__lowercase=True, to_idx__ngram_range=(2, 2), to_idx__stop_words=None, total=   7.2s\n",
      "[CV] net__lr=0.00159506240923, net__max_epochs=5, net__module__dropout=0.523145585372, net__module__embedding_dim=181, net__module__num_layers=3, net__module__num_units=57, net__module__rec_layer_type=gru, to_idx__lowercase=False, to_idx__ngram_range=(2, 2), to_idx__stop_words=english \n",
      "[CV]  net__lr=0.00159506240923, net__max_epochs=5, net__module__dropout=0.523145585372, net__module__embedding_dim=181, net__module__num_layers=3, net__module__num_units=57, net__module__rec_layer_type=gru, to_idx__lowercase=False, to_idx__ngram_range=(2, 2), to_idx__stop_words=english, total=   7.1s\n",
      "[CV] net__lr=0.00159506240923, net__max_epochs=5, net__module__dropout=0.523145585372, net__module__embedding_dim=181, net__module__num_layers=3, net__module__num_units=57, net__module__rec_layer_type=gru, to_idx__lowercase=False, to_idx__ngram_range=(2, 2), to_idx__stop_words=english \n",
      "[CV]  net__lr=0.00159506240923, net__max_epochs=5, net__module__dropout=0.523145585372, net__module__embedding_dim=181, net__module__num_layers=3, net__module__num_units=57, net__module__rec_layer_type=gru, to_idx__lowercase=False, to_idx__ngram_range=(2, 2), to_idx__stop_words=english, total=   6.4s\n",
      "[CV] net__lr=0.00159506240923, net__max_epochs=5, net__module__dropout=0.523145585372, net__module__embedding_dim=181, net__module__num_layers=3, net__module__num_units=57, net__module__rec_layer_type=gru, to_idx__lowercase=False, to_idx__ngram_range=(2, 2), to_idx__stop_words=english \n",
      "[CV]  net__lr=0.00159506240923, net__max_epochs=5, net__module__dropout=0.523145585372, net__module__embedding_dim=181, net__module__num_layers=3, net__module__num_units=57, net__module__rec_layer_type=gru, to_idx__lowercase=False, to_idx__ngram_range=(2, 2), to_idx__stop_words=english, total=   6.3s\n",
      "[CV] net__lr=0.0794441043086, net__max_epochs=10, net__module__dropout=0.207479721156, net__module__embedding_dim=136, net__module__num_layers=1, net__module__num_units=153, net__module__rec_layer_type=gru, to_idx__lowercase=True, to_idx__ngram_range=(2, 2), to_idx__stop_words=None \n",
      "[CV]  net__lr=0.0794441043086, net__max_epochs=10, net__module__dropout=0.207479721156, net__module__embedding_dim=136, net__module__num_layers=1, net__module__num_units=153, net__module__rec_layer_type=gru, to_idx__lowercase=True, to_idx__ngram_range=(2, 2), to_idx__stop_words=None, total=  10.4s\n",
      "[CV] net__lr=0.0794441043086, net__max_epochs=10, net__module__dropout=0.207479721156, net__module__embedding_dim=136, net__module__num_layers=1, net__module__num_units=153, net__module__rec_layer_type=gru, to_idx__lowercase=True, to_idx__ngram_range=(2, 2), to_idx__stop_words=None \n",
      "[CV]  net__lr=0.0794441043086, net__max_epochs=10, net__module__dropout=0.207479721156, net__module__embedding_dim=136, net__module__num_layers=1, net__module__num_units=153, net__module__rec_layer_type=gru, to_idx__lowercase=True, to_idx__ngram_range=(2, 2), to_idx__stop_words=None, total=  10.2s\n",
      "[CV] net__lr=0.0794441043086, net__max_epochs=10, net__module__dropout=0.207479721156, net__module__embedding_dim=136, net__module__num_layers=1, net__module__num_units=153, net__module__rec_layer_type=gru, to_idx__lowercase=True, to_idx__ngram_range=(2, 2), to_idx__stop_words=None \n",
      "[CV]  net__lr=0.0794441043086, net__max_epochs=10, net__module__dropout=0.207479721156, net__module__embedding_dim=136, net__module__num_layers=1, net__module__num_units=153, net__module__rec_layer_type=gru, to_idx__lowercase=True, to_idx__ngram_range=(2, 2), to_idx__stop_words=None, total=  10.4s\n",
      "[CV] net__lr=1.26492960154e-06, net__max_epochs=5, net__module__dropout=0.345116002531, net__module__embedding_dim=73, net__module__num_layers=3, net__module__num_units=72, net__module__rec_layer_type=gru, to_idx__lowercase=True, to_idx__ngram_range=(2, 2), to_idx__stop_words=english \n",
      "[CV]  net__lr=1.26492960154e-06, net__max_epochs=5, net__module__dropout=0.345116002531, net__module__embedding_dim=73, net__module__num_layers=3, net__module__num_units=72, net__module__rec_layer_type=gru, to_idx__lowercase=True, to_idx__ngram_range=(2, 2), to_idx__stop_words=english, total=   5.8s\n",
      "[CV] net__lr=1.26492960154e-06, net__max_epochs=5, net__module__dropout=0.345116002531, net__module__embedding_dim=73, net__module__num_layers=3, net__module__num_units=72, net__module__rec_layer_type=gru, to_idx__lowercase=True, to_idx__ngram_range=(2, 2), to_idx__stop_words=english \n",
      "[CV]  net__lr=1.26492960154e-06, net__max_epochs=5, net__module__dropout=0.345116002531, net__module__embedding_dim=73, net__module__num_layers=3, net__module__num_units=72, net__module__rec_layer_type=gru, to_idx__lowercase=True, to_idx__ngram_range=(2, 2), to_idx__stop_words=english, total=   5.9s\n",
      "[CV] net__lr=1.26492960154e-06, net__max_epochs=5, net__module__dropout=0.345116002531, net__module__embedding_dim=73, net__module__num_layers=3, net__module__num_units=72, net__module__rec_layer_type=gru, to_idx__lowercase=True, to_idx__ngram_range=(2, 2), to_idx__stop_words=english \n",
      "[CV]  net__lr=1.26492960154e-06, net__max_epochs=5, net__module__dropout=0.345116002531, net__module__embedding_dim=73, net__module__num_layers=3, net__module__num_units=72, net__module__rec_layer_type=gru, to_idx__lowercase=True, to_idx__ngram_range=(2, 2), to_idx__stop_words=english, total=   5.9s\n",
      "[CV] net__lr=0.000394032773719, net__max_epochs=10, net__module__dropout=0.516892723965, net__module__embedding_dim=76, net__module__num_layers=1, net__module__num_units=157, net__module__rec_layer_type=gru, to_idx__lowercase=True, to_idx__ngram_range=(1, 1), to_idx__stop_words=None \n",
      "[CV]  net__lr=0.000394032773719, net__max_epochs=10, net__module__dropout=0.516892723965, net__module__embedding_dim=76, net__module__num_layers=1, net__module__num_units=157, net__module__rec_layer_type=gru, to_idx__lowercase=True, to_idx__ngram_range=(1, 1), to_idx__stop_words=None, total=   7.7s\n",
      "[CV] net__lr=0.000394032773719, net__max_epochs=10, net__module__dropout=0.516892723965, net__module__embedding_dim=76, net__module__num_layers=1, net__module__num_units=157, net__module__rec_layer_type=gru, to_idx__lowercase=True, to_idx__ngram_range=(1, 1), to_idx__stop_words=None \n",
      "[CV]  net__lr=0.000394032773719, net__max_epochs=10, net__module__dropout=0.516892723965, net__module__embedding_dim=76, net__module__num_layers=1, net__module__num_units=157, net__module__rec_layer_type=gru, to_idx__lowercase=True, to_idx__ngram_range=(1, 1), to_idx__stop_words=None, total=   7.7s\n",
      "[CV] net__lr=0.000394032773719, net__max_epochs=10, net__module__dropout=0.516892723965, net__module__embedding_dim=76, net__module__num_layers=1, net__module__num_units=157, net__module__rec_layer_type=gru, to_idx__lowercase=True, to_idx__ngram_range=(1, 1), to_idx__stop_words=None \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  net__lr=0.000394032773719, net__max_epochs=10, net__module__dropout=0.516892723965, net__module__embedding_dim=76, net__module__num_layers=1, net__module__num_units=157, net__module__rec_layer_type=gru, to_idx__lowercase=True, to_idx__ngram_range=(1, 1), to_idx__stop_words=None, total=   7.8s\n",
      "CPU times: user 3min 55s, sys: 53.1 s, total: 4min 48s\n",
      "Wall time: 4min 48s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done  30 out of  30 | elapsed:  4.8min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "RandomizedSearchCV(cv=3, error_score='raise',\n",
       "          estimator=Pipeline(memory=None,\n",
       "     steps=[('to_idx', TextFeaturizer(analyzer='word', binary=False, decode_error='strict',\n",
       "        dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',\n",
       "        lowercase=True, max_df=1.0, max_features=1000, min_df=1,\n",
       "        ngram_range=(1, 1), preprocessor=None, stop_words=None,\n",
       "        stri...ayers=2, batch_first=True)\n",
       "    (output): Linear(in_features=128, out_features=2, bias=True)\n",
       "  ),\n",
       "))]),\n",
       "          fit_params=None, iid=True, n_iter=10, n_jobs=1,\n",
       "          param_distributions={'to_idx__stop_words': ['english', None], 'to_idx__lowercase': [False, True], 'to_idx__ngram_range': [(1, 1), (2, 2)], 'net__module__embedding_dim': <scipy.stats._distn_infrastructure.rv_frozen object at 0x7f60dbc226d8>, 'net__module__rec_layer_type': ['gru', 'lstm'], 'net__modul... 1.2649296015411273e-06, 0.0015950624092266551, 0.00039403277371927766], 'net__max_epochs': [5, 10]},\n",
       "          pre_dispatch='2*n_jobs', random_state=None, refit=False,\n",
       "          return_train_score=True, scoring='accuracy', verbose=2)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%time search.fit(X[:5000], y[:5000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below we see the best accuracy we achieved and what the best hyper-parameters were. Of course the scores here are underwhelming, given that we used so few samples and iterations. Using all the data and trying out more iterations should lead to much better outcomes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.6784,\n",
       " {'net__lr': 0.00039403277371927766,\n",
       "  'net__max_epochs': 10,\n",
       "  'net__module__dropout': 0.51689272396462094,\n",
       "  'net__module__embedding_dim': 76,\n",
       "  'net__module__num_layers': 1,\n",
       "  'net__module__num_units': 157,\n",
       "  'net__module__rec_layer_type': 'gru',\n",
       "  'to_idx__lowercase': True,\n",
       "  'to_idx__ngram_range': (1, 1),\n",
       "  'to_idx__stop_words': None})"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "search.best_score_, search.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
